Compressive Sense Imaging

ABSTRACT

Systems and methods for compressive sense imaging are provided. In one aspect, incident light reflecting from an object is received via an aperture array and a sensor and intermediate compressive measurements are generated using compressive sequence matrices that are determined based on the properties of the aperture array and the sensor. The intermediate compressive measurements are further processed to generate compressive measurements representing the compressed image of the object. An uncompressed image of the object is generated from the compressive measurements using a determined reconstruction matrix that is different from the sequence matrices used to acquire the intermediate compressive measurements.

CROSS-REFERENCE

The present application references subject matter of the following U.S.applications, each of which is incorporated by reference herein in itsentirety: U.S. application Ser. No. 13/658,904 filed on Oct. 24, 2012and entitled “Resolution and Focus Enhancement”; U.S. application Ser.No. 13/658,900 filed on Oct. 24, 2012 and entitled “Lensless CompressiveImage Acquisition”; U.S. application Ser. No. 13/367,413 filed on Feb.7, 2012 and entitled “Lensless Compressive Image Acquisition”; and, U.S.application Ser. No. 12/894,855 filed on Sep. 30, 2010 and entitled“Apparatus and Method for Generating Compressive Measurements of VideoUsing Spatial and Temporal Integration”, which issued as U.S. Pat. No.8,644,376 on Feb. 4, 2014.

TECHNICAL FIELD

This disclosure is directed to systems and methods for compressive senseimage processing.

BACKGROUND

This section introduces aspects that may be helpful in facilitating abetter understanding of the systems and methods disclosed herein.Accordingly, the statements of this section are to be read in this lightand are not to be understood or interpreted as admissions about what isor is not in the prior art.

Digital image/video cameras acquire and process a significant amount ofraw data. In order to store or transmit image data efficiently, the rawpixel data for each of the N pixels of an N-pixel image is firstcaptured and then typically compressed using a suitable compressionalgorithm for storage and/or transmission. Although compression aftercapturing the raw data for each of the N pixels of the image isgenerally useful for reducing the size of the image (or video) capturedby the camera, it requires significant computational resources and time.In addition, compression of the raw pixel data does not alwaysmeaningfully reduce the size of the captured images.

A more recent approach, known as compressive sense imaging, acquirescompressed image (or video) data using random projections without firstcollecting the raw data for all of the N pixels of an N-pixel image. Forexample, a compressive measurement basis is applied to obtain a seriesof compressive measurements which represent the encoded (i.e.,compressed) image. Since a reduced number of compressive measurementsare acquired in comparison to the raw data for each of the N pixelvalues of a desired N-pixel image, this approach can significantlyeliminate or reduce the need for applying compression after the raw datais captured.

BRIEF SUMMARY

Systems and methods for compressive sense imaging are provided. In someembodiments, incident light reflecting from an object and passingthrough an aperture array is detected by a sensor. Intermediatecompressive measurements are generated based on the output by the sensorusing compressive sequence matrices that are determined based on theproperties of the aperture array and the sensor. The intermediatecompressive measurements are further processed to generate compressivemeasurements representing the compressed image of the object. Anuncompressed image of the object is generated from the compressivemeasurements using a determined reconstruction matrix that is differentfrom the sequence matrices used to acquire the intermediate compressivemeasurements.

In one aspect, a compressive sense imaging system and method includesgenerating a plurality of sequence matrices; determining a plurality ofintermediate compressive measurements using the plurality of sequencematrices; and, generating a plurality of compressive measurementsrepresenting a compressed image of an object using the plurality ofintermediate compressive measurements.

In some aspects, the system and method includes generating anuncompressed image of the object from the plurality of compressivemeasurements using a reconstruction basis matrix.

In some aspects, the system and method includes determining a kernelmatrix based on properties of an aperture array of aperture elements anda sensor, and, generating a sensing matrix using the kernel matrix and areconstruction basis matrix.

In some aspects, the system and method includes decomposing the sensingmatrix to generate the plurality of sequence matrices.

In some aspects, the system and method includes determining asensitivity function for the sensor;

determining at least one characteristic function for at least one of theaperture elements of the aperture array; computing a kernel function byperforming a convolution operation using the sensitivity function andthe at least one characteristic function; and, determining the kernelmatrix using the kernel function and an image.

In some aspects, the system and method includes applying a sparsifyingoperator to generate the uncompressed image of the object from theplurality of compressive measurements using the reconstruction basismatrix.

In some aspects, the system and method includes selectively enabling ordisabling one or more aperture elements of an aperture array based on atleast one basis in a sequence matrix to determine at least one of theplurality of intermediate compressive measurements during a time period,where the at least one of the plurality of intermediate compressivemeasurements is determined based on an aggregated sum of light detectedby the sensor during the time period.

In some aspects, the aperture array is an array of micro-mirrors. Insome aspects, the aperture array is an array of LCD elements.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a compressive sense imaging system inaccordance with various aspects of the disclosure.

FIG. 2 illustrates an example of a camera unit for acquiring compressivemeasurements of an object using a sequence matrix in accordance with oneaspect of the disclosure.

FIG. 3 illustrates an example process for compressive sense imaging inaccordance with various aspects of the disclosure.

FIG. 4 illustrates an example apparatus for implementing aspects of thedisclosure.

DETAILED DESCRIPTION

Various aspects of the disclosure are described below with reference tothe accompanying drawings, in which like numerals refer to like elementsin the description of the figures. The description and drawings merelyillustrate the principles of the disclosure; various structures, systemsand devices are described and depicted in the drawings for purposes ofexplanation only and so as not to obscure the present invention withdetails that are well known to those skilled in the art, who will beable to devise various arrangements that, although not explicitlydescribed or shown herein, embody the principles and are included withinspirit and scope of the disclosure.

As used herein, the term, “or” refers to a non-exclusive or, unlessotherwise indicated (e.g., “or else” or “or in the alternative”).Furthermore, words used to describe a relationship between elementsshould be broadly construed to include a direct relationship or thepresence of intervening elements unless otherwise indicated. Forexample, when an element is referred to as being “connected” or“coupled” to another element, the element may be directly connected orcoupled to the other element or intervening elements may be present. Incontrast, when an element is referred to as being “directly connected”or “directly coupled” to another element, there are no interveningelements present. Similarly, words such as “between”, “adjacent”, andthe like should be interpreted in a like fashion.

The singular forms “a”, “an”, and “the” are intended to include theplural forms as well, unless the context clearly indicates otherwise. Itwill be further understood that the terms “comprises”, “comprising,”,“includes” and “including”, when used herein, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

FIG. 1 illustrates a schematic example of a compressive imagingacquisition and reconstruction system 100 (“system 100”). Incident light105 reflecting from an object 110 is received by the camera unit 115,which generates a plurality of intermediate compressive measurementsusing a determined number of compressive sequence matrices 120. Theintermediate compressive measurements are further processed to generatecompressive measurements 125 representing the compressed image of theobject 110. The compressive measurements 125 representing the compressedimage of the object 110 may be stored (or transmitted) by astorage/transmission unit 130. The reconstruction unit 135 generates anuncompressed image 140 (e.g., for display on a display unit) of theobject 110 from the compressive measurements 125 using a determinedreconstruction matrix 150.

Although the units are shown separately in FIG. 1, this is merely to aidunderstanding of the disclosure. In other aspects the functionality ofany or all of the units described above may be implemented using feweror greater number of units. Furthermore, the functionality attributed tothe various units may be implemented by a single processing device ordistributed amongst multiple processing devices. Some examples ofsuitable processing devices include cameras, camera systems, mobilephones, personal computer systems, tablets, set-top boxes, smart phonesor any type of computing device configured to acquire, process, oroutput data.

In one embodiment, a single processing device may be configured toprovide the functionality of each of the units of system 100. The singleprocessing device may include, for example, a memory storing one or moreinstructions, and a processor for executing the one or moreinstructions, which, upon execution, may configure the processor toprovide functionality ascribed to the units. The single processingdevice may include other components typically found in computingdevices, such as one or more input/output components for inputting oroutputting information to/from the processing device, including acamera, a display, a keyboard, a mouse, network adapter, etc.

In another embodiment, a local processing device may be provided at afirst location that is communicatively interconnected with a remoteprocessing device at a remote location via network. The local processingdevice may be configured with the functionality to generate and providethe compressive measurements 125 of the local object 110 to a remoteprocessing device over the network. The remote processing device, inturn, may be configured to receive the compressive measurements from thelocal processing device, to generate the reconstructed image 140 fromthe compressive measurements 125 using the reconstruction basis matrix150, and to display the reconstructed image to a remote user inaccordance with the aspects described below. The local processing deviceand the remote processing device may be respectively implemented usingan apparatus similar to the single processing device, and may include amemory storing one or more instructions, a processor for executing theone or more instructions, and various input/output components as in thecase of the single processing device. The network may be an intranet,the Internet, or any type or combination of one or more wired orwireless networks.

FIG. 2 illustrates an example of a lensless camera unit 115 foracquiring compressive measurements 125 representing the compressed imageof the object 110 using compressive sense imaging. Although a particularembodiment of the lensless camera unit 115 is described, this is not tobe construed as a limitation, and the principles of the disclosure maybe applied to other embodiments of compressive sense imaging systems.

Incident light 105 reflected off the object 110 is received at thecamera unit 115 where the light 105 is selectively permitted to passthrough an aperture array 220 of N individual aperture elements andstrike a sensor 230. The camera unit 115 processes the output of thesensor 230 to produce intermediate compressive measurements using aplurality of sequence matrices that are determined based on one or moreproperties of the aperture array 220 and the sensor 230. The compressivemeasurements 125 collectively represent the compressed image of theobject 110 and are determined using the intermediate compressivemeasurements.

To achieve compression, the number M of the compressive measurements 125that are acquired as the compressed image of the object 110 is typicallysignificantly less than the N raw data values that are acquired in aconventional camera system having an N-pixel sensor for generating anN-pixel image, thus reducing or eliminating the need for conventionalcompression of the raw data values after acquisition. In practice, thenumber of compressive measurements M may be pre-selected relative to theN aperture elements of the array 220 based upon a desired balancebetween the level of compression and the quality of the N-pixel image140 that is reconstructed using the M compressive measurements.

The example array 220 illustrated in FIG. 2 is a two dimensional, 8×8array of sixty-four (N=64) discrete aperture elements, which arearranged in two dimensional row and column format such that individualelements of the array 220 may be uniquely identified using a tabularnotation form “[row, column]”. Thus, the first element in the first rowof array 220 is exemplarily referenced as 220[1,1], and the last elementin the last row of the array 220 is referenced as 220[8,8].

In practice, the size and format of the array 220 may have asignificantly greater (or fewer) number of elements, depending on thedesired resolution of the image 140. By way of example only, the array220 may be a 640×480 (N=307,200) element array for a desired imageresolution of 640×480 pixels for the image 140, or may be a 1920×1080(N=2,073,600) element array for a correspondingly greater desiredresolution of the image 140.

The overall transmittance of light 105 passing through the array 220 andreaching the sensor 230 at a given time may be varied by setting thetransmittance of one or more of the individual aperture elements of thearray. For example, the overall transmittance of array 220 may beadjusted by selectively and individually changing the transmittance ofone or more of the aperture elements 220[1,1] to 220[8,8] to increase ordecrease the amount of light 105 passing through the array 220 andreaching the sensor 230 at a given time.

Aperture elements that are fully opened (e.g., fully enabled oractivated) allow light 105 to pass through those opened elements andreach the sensor 230, whereas aperture elements that are fully closed(e.g., fully disabled or deactivated) prevent or block light 105 frompassing through the closed elements of the array 220 and reaching thephoton detector 230. The aperture elements may be partially opened (orpartially closed) to pass only some, but not all, of the light 105 toreach the sensor 230 via the partially opened (or partially closed)elements. Thus, the collective state of the individual aperture elements(e.g., opened, closed, or partially opened or closed) determines theoverall transmittance of the aperture array 220 and therefore determinesthe amount of light 105 reaching the sensor 230 at a given time.

In one embodiment, the aperture array 220 is a micro-mirror array of Nindividually selectable micro-mirrors. In another embodiment, theaperture array 120 may be an N element LCD array. In other embodiments,the aperture array 220 may be any suitable array of electronic oroptical components having selectively controllable transmittance.

The camera unit 115 is configured to generate intermediate compressivemeasurements by selectively adjusting the overall transmittance of theaperture array 220 in accordance with compressive bases information in aplurality of sequence matrices. Each of the intermediate compressivemeasurements may be understood as the determined sum (or aggregate) ofthe light 105 reaching the sensor 230 through the array 220 during aparticular time when particular ones of the N aperture elements of thearray 220 are selectively opened and closed (either fully or partially)in accordance with a pattern indicated by a particular compressive basisof a sequence matrix 120.

One feature of the present disclosure is that a M number of intermediatecompressive measurements are acquired using each of a S number ofsequence matrices that are determined as described further below. SinceS≧2, at least 2M number of intermediate compressive measurements aredetermined, which are processed into M compressive measurements 125representing the compressed image of the object 110 as described furtherbelow. The M compressive measurements 125 are used in conjunction withthe reconstruction matrix 150 to reconstruct or generate theuncompressed image 140 of the object 110. Another feature of the presentdisclosure is that the sequence matrices are determined based on akernel function, where the kernel function is determined based on theproperties of the array 220 and the sensor 230. These and other aspectsof the present disclosure are described in detail further below.

In general, a determined sequence matrix 120 is a set of M compressivebases b₁, b₂, . . . b_(M), each of which is applied in turn to the array220 to produce a respective one of M intermediate compressivemeasurements. Each measurement basis b₁, b₂, . . . b_(M) in the sequencematrix 120 is itself an array of N values corresponding to the number Nof aperture elements of the array 220, as indicated mathematicallybelow:

$\begin{bmatrix}{b_{1}\lbrack 1\rbrack} & {b_{1}\lbrack 2\rbrack} & \ldots & {b_{1}\lbrack N\rbrack} \\{b_{2}\lbrack 1\rbrack} & {b_{2}\lbrack 2\rbrack} & \ldots & {b_{2}\lbrack N\rbrack} \\{b_{3}\lbrack 1\rbrack} & {b_{3}\lbrack 2\rbrack} & \ldots & {b_{3}\lbrack N\rbrack} \\\vdots & \vdots & \vdots & \vdots \\{b_{M}\lbrack 1\rbrack} & {b_{M}\lbrack 2\rbrack} & \ldots & {b_{M}\lbrack N\rbrack}\end{bmatrix}\quad$

For example, in the embodiment illustrated in FIG. 2, each compressivebasis b_(k)(kε[1 . . . M]) of a given sequence matrix 120 is a set ofvalues b_(k) [1] to b_(k) [64] where each value is normalized to a set[0,1] as described later below. Accordingly, each value of a givencompressive basis may be a “0”, “1”, or a real value between “0” and“1”, which respectively determines the corresponding state (e.g., fullyclosed, fully opened, or a state in-between) of a respective apertureelement in the 8×8 aperture array 220.

A given compressive basis b_(k) is applied to the array 220 to produce acorresponding intermediate compressive measurement for a time t_(k) asfollows. The respective values b_(k)[1] to b_(k)[64] are used to set thestate (fully opened, fully closed or partially opened or closed) of thecorresponding elements of array 220, and the detected sum or aggregateof light 105 reaching the sensor 230 is determined as the value of thecorresponding intermediate compressive measurement. A total number ofM×S intermediate compressive measurements are produced in this manner,where M is the number of compressive bases in each sequence matrix 120and S is the number of sequence matrices (where S≧2).

An example operation of system 100 is now described in conjunction withthe process 300 of FIG. 3. As an overview for aiding the reader, steps302-308 describe the determination of the sequence matrices 120. Step310 describes determination of the compressive measurements 125representing the compressed image of the object 110 from theintermediate compressive measurements acquired using the sequencematrices 120. Step 312 describes generating the uncompressed image ofthe object 110 from the compressive measurements 125 using thereconstruction matrix 150.

It is to be understood that the steps described below are merelyillustrative and that existing steps may be modified or omitted,additional steps may be added, and the order of certain steps may bealtered.

Turning now to the process 300 of FIG. 3, the determination of thesequence matrices 120 begins in step 302 with the computation of a N×Nkernel matrix K that is determined based on the geometry and propertiesof the array 220 and the sensor 230. The kernel matrix may be determinedas follows.

In one embodiment, the kernel matrix is computed based on a sensitivityfunction for the sensor 230 and a characteristic function of the array220. First, a sensitivity function F(x,y) of the sensor 230 isdetermined, where F(x,y) is the response of the sensor 230 when lightstrikes a point x,y on the sensor in Cartesian coordinates. Preferably,but not necessarily, the sensor 230 is selected such that it has a largesensing area and a uniform (or close to uniform) sensitivity functionF(x,y), such that the sensor response (or, in other words, sensorsensitivity) does not vary (or does not vary very much) based on thewhere the light strikes the sensor.

Next, a characteristic function for each of the aperture elements of thearray is defined, such that the characteristic function E(x,y) of agiven aperture element E is E(x,y)=1 if a point x,y in Cartesiancoordinates falls within the area of the aperture element and E(x,y)=0if the point x,y in lies outside the area of the aperture element.

Next, a kernel function k(x,y) is defined using the sensitivity functionof the sensor 230 and the characteristic function of the apertureelements of the array 220 as k(x,y)=E*F, where, the * operator indicatestwo-dimensional (2D) convolution operation. A discrete kernel functionk(row,column) is determined as:

k(row,column)=∫∫_(E) _(row,column) k(x,y)dxdy,

where E_(row,column) identifies a particular aperture element E of thearray 220 using the row, column notation.

It is noted here that alternatively, in another embodiment, the discretekernel function may also be obtained by calibrating the camera unit 115using a point lighting source (e.g., a laser source or another lightingsource that is in effect a point lighting source with respect to thecamera unit 115).

Finally, the N×N kernel matrix K is computed from the discrete kernelfunction as:

K·I _(1D)=(k(row,column)*I _(2D))_(1D),

where 1D indicates the one-dimensional (1D) vector form of a 2D array,and I is any N-pixel image.

In step 304, the determination of the sequence matrices 120 continues byspecifying the reconstruction matrix 150. The reconstruction matrix maybe any M×N matrix that has a property suitable for use in compressivesense imaging, such as, for example, the Restricted Isometry Property.In one embodiment, accordingly, the reconstruction matrix 150 is a M×Nmatrix whose rows are selected from randomly or pseudo-randomly permutedN×N Hadamard matrix, having the known properties that the entries orvalues of such reconstruction matrix are either +1 or −1 and the rowsare mutually orthogonal.

In step 306, the determination of the sequence matrices 120 continues bycomputing a M×N sensing matrix A, where the sensing matrix is computedas:

A=[α _(ij) ]=RK ⁻¹

where, R is the M×N reconstruction matrix computed in step 304 and K⁻¹is the N×N inverse matrix of the N×N kernel matrix K that was determinedin step 302 based on the properties of the sensor 230 and the array 220,and where [α_(ij)] are the values of the sensing matrix A for i=1, . . .M and j=1, . . . N.

It is pointed out that while sensing matrix A is a M×N matrix that isdetermined based on the properties of the array 220 and the sensor 230,it is not suitable for use as a sequence matrix 120 directly. This isbecause, as will be apparent at least from the negatively values of thereconstruction matrix R, one or more of values [α_(ij)] of the sensingmatrix A do not satisfy 0≦α_(ij)≦1. In fact, the sensing matrix A mayinclude large negative and positive values, which are impractical (orperhaps not possible) to use as a pattern for setting the condition ofthe aperture elements of the array 220.

As a result, in step 308, the sensing matrix A is further decomposedinto the sequence matrices 120 that have values that are within the set[0,1] as follows. It is also noted that while the description below isprovided for the sequence matrices to have values within the set [0,1],the disclosure below is applicable to decomposing the sensing matrix Ato have values within other sets.

Given the sensing matrix A, define:

${A^{+} = \left\lbrack a_{i,j}^{+} \right\rbrack},{{{where}\mspace{14mu} a_{i,j}^{+}} = \left\{ {{\begin{matrix}{a_{i,j},} & {{{for}\mspace{14mu} a_{i,j}} > 0} \\{{0,}\mspace{20mu}} & {{{for}\mspace{14mu} a_{i,j}} < 0}\end{matrix}\mspace{14mu} {and}},{A^{-} = \left\lbrack a_{i,j}^{-} \right\rbrack},{{{where}\mspace{14mu} a_{i,j}^{-}} = \left\{ \begin{matrix}{- a_{i,j}} & {{{for}\mspace{14mu} a_{i,j}} < 0} \\{{0,}\mspace{31mu}} & {{{for}\mspace{14mu} a_{i,j}} \geq 0}\end{matrix} \right.}} \right.}$

for i=1, . . . M and j=1, . . . N.

Next, A⁺ is decomposed into a P⁺ number of M×N sequence matrices A_(k)⁺=[α_(i,j) ^((k)+)] where, i=1, . . . M, j=1, . . . N, and k=1, . . . ,P³⁺ using the following pseudo-code algorithm:

   for i = 1, . . . M, j = 1, . . . , N   let p = 0, a_(ij) ⁽⁰⁾⁺ = 0   ${{{while}\mspace{14mu} a_{ij}^{+}} - {\sum\limits_{k = 1}^{p}a_{ij}^{{(k)} +}}} > 1$   $a_{ij}^{{({p + 1})} +} = {{clip}\left( {{a_{ij}^{+} - {\sum\limits_{k = 1}^{p}a_{ij}^{{(k)} +}}},1} \right)}$   p ← p + 1   end   $a_{ij}^{{({p + 1})} +} = {a_{ij}^{+} - {\sum\limits_{k = 1}^{p}a_{ij}^{{(k)} +}}}$  P⁺ (i, j) = p + 1  end  ${{define}\mspace{14mu} P^{+}} = {\max\limits_{i,j}\mspace{11mu} {P\left( {i,j} \right)}}$ for k = 1, . . . , P⁺   A_(k) ⁺ = [a_(ij) ^((k)+)], where a_(ij)^((k)+) = 0 if k > P⁺ (i, j)${where},{{{clip}\left( {x,u} \right)} = \left\{ \begin{matrix}{x,} & {{{if}\mspace{14mu} 0} \leq x \leq µ} \\{µ,} & {otherwise}\end{matrix} \right.}$

Next, matrix A⁻ may be similarly decomposed into a P⁻ number of M×Nsequence matrices A_(k) ⁻=[α_(i,j) ^((k)−)] where, i=1, . . . M, j=1, .. . N, and k=1, . . . , P⁻ based on the algorithm above.

It is noted that all of the values of the resulting P⁺ number of M×Nsequence matrices A_(k) ⁺=[α_(i,j) ^((k)+)] satisfy 0≦α_(ij) ⁺≦1, and,similarly, all of the values of the each of the resulting P⁻ number ofM×N sequence matrices A_(k) ⁻=[α_(i,j) ^((k)−)] also satisfy 0≦α_(ij)⁻≦1.

The decomposition of the sensing matrix into the sequence matricesdescribed above leads to the equation:

$A = {{\sum\limits_{k = 1}^{P^{+}}\; A_{k}^{+}} - {\sum\limits_{k = 1}^{P^{-}}\; A_{k}^{-}}}$

In step 310, each of the determined sequence matrices A_(k) ⁺ and A_(k)⁻ are applied to the array 220 to acquire the intermediate compressivemeasurements as described previously. For example, in one embodiment,each M×N sequence matrix A_(k) ⁺ (k=1, . . . , P⁺) is applied to thearray 220 to generate a measurement vector y_(k) ⁺ of the correspondingset of M intermediate compressive measurements. Similarly, each M×Nsequence matrix A_(k) ⁻ (k=1, . . . , P⁻) is also applied to the array220 to generate a measurement vector y_(k) ⁻ of the corresponding set ofM intermediate compressive measurements.

In step 312, the process includes determining the compressivemeasurements 125 representing the compressed image of the object 110from the intermediate compressive measurements determined in step 310,and reconstructing the uncompressed image of the object 110 from thecompressive measurements 125.

In particular, the M number of compressive measurements 125 aredetermined using the intermediate compressive measurements y_(k) ⁺ andy_(k) ⁻ as:

$y = {{\sum\limits_{k = 1}^{P^{+}}\; y_{k}^{+}} - {\sum\limits_{k = 1}^{P^{-}}\; y_{k}^{-}}}$

The uncompressed image I of the object 110 may be determined using thecompressive measurements 125 and the reconstruction matrix 150 as:

min∥W·I∥₁, subject to: R·I=y=Σ_(k=1) ^(P+)y_(k) ⁺−Σ_(k=1) ^(P−)y_(k) ⁻

where W is a sparsifying operator, I is the one-dimensional matrixrepresentation of the N valued image 140, R is the reconstruction basismatrix determined in step 304, and y=Y₁, Y₂, Y₃ . . . Y_(M) is a columnvector of the compressive measurements 125 acquired based on theintermediate compressive measurements acquired using the sequencematrices. The sparsifying operator W may be generated, for example, byusing wavelets, or by using total variations.

Steps 304 to 312 of the process described above may be repeated orperformed once per image or video frame. Step 302 need not be repeatedunless a different kernel matrix K is desired, for example, if there isa change in the array 220 or the sensor 230.

The present disclosure is believed to incur a number of advantages. Tobegin with, it describes an improved lensless camera unit suitable forcompressive sense imaging that provides better images in low-lighthaving a higher signal-to-noise ratio due to a larger number ofmeasurements (at least 2×M) acquired using the array 220 to produce theM number of compressive measurements. In addition, the measurements areacquired in a manner that takes particular properties of the aperturearray and the sensor into account. To continue, the present disclosureis suited for images in all spectrum of light, including the visible andthe invisible spectrum. In addition, the present disclosure alsoprovides for capturing images that sharper (e.g., having a greateramount of detail) for a given sensor geometry and size, and particularlyfor sensor and aperture arrays that are relatively large, which areotherwise known to produce soft (relatively blurrier) images.

It will be appreciated that one or more aspects of the disclosure may beimplemented using hardware, software, or a combination thereof. FIG. 4depicts a high-level block diagram of an example processing device orapparatus 400 suitable for implementing one or more aspects of thedisclosure. Apparatus 400 comprises a processor 402 that iscommunicatively interconnected with various input/output devices 404 anda memory 406.

The processor 402 may be any type of processor such as a general purposecentral processing unit (“CPU”) or a dedicated microprocessor such as anembedded microcontroller or a digital signal processor (“DSP”). Theinput/output devices 404 may be any peripheral device operating underthe control of the processor 402 and configured to input data into oroutput data from the apparatus 400 in accordance with the disclosure,such as, for example, a lens or lensless camera or video capture devicewhich may include a aperture array and a sensor. The input/outputdevices 404 may also include conventional network adapters, data ports,and various user interface devices such as a keyboard, a keypad, amouse, or a display.

Memory 406 may be any type of memory suitable for storing electronicinformation, including data and instructions executable by the processor402. Memory 406 may be implemented as, for example, as one or morecombinations of a random access memory (RAM), read only memory (ROM),flash memory, hard disk drive memory, compact-disk memory, opticalmemory, etc. In addition, apparatus 400 may also include an operatingsystem, queue managers, device drivers, or one or more network protocolswhich may be stored, in one embodiment, in memory 406 and executed bythe processor 402.

The memory 406 may include non-transitory memory storing executableinstructions and data, which instructions, upon execution by theprocessor 402, may configure apparatus 400 to perform the functionalityin accordance with the various aspects and steps described above. Insome embodiments, the processor 402 may be configured, upon execution ofthe instructions, to communicate with, control, or implement all or apart of the functionality with respect to the acquisition or thereconstruction of the compressive measurements as described above. Theprocessor may be configured to determine the sequence matrices, theintermediate compressive measurements, the compressive measurements, andto generate the uncompressed images or video using a determinedreconstruction matrix as described above.

In some embodiments, the processor 402 may also be configured tocommunicate with and/or control another apparatus 400 to which it isinterconnected via, for example a network. In such cases, thefunctionality disclosed herein may be integrated into each standaloneapparatus 400 or may be distributed between one or more apparatus 400.In some embodiments, the processor 402 may also be configured as aplurality of interconnected processors that are situated in differentlocations and communicatively interconnected with each other (e.g., in acloud computing environment).

While a particular apparatus configuration is shown in FIG. 4, it willbe appreciated that the present disclosure not limited to any particularimplementation. For example, in some embodiments, all or a part of thefunctionality disclosed herein may be implemented using one or moreapplication specific integrated circuits (ASICs), field programmablegate arrays (FPGAs), or the like.

Although aspects herein have been described with reference to particularembodiments, it is to be understood that these embodiments are merelyillustrative of the principles and applications of the presentdisclosure. It is therefore to be understood that numerous modificationscan be made to the illustrative embodiments and that other arrangementscan be devised without departing from the spirit and scope of thedisclosure.

1. A compressive sense imaging system, the system comprising: aprocessing device configured to: generate a plurality of sequencematrices; determine a plurality of intermediate compressive measurementsusing the plurality of sequence matrices; and, generate a plurality ofcompressive measurements representing a compressed image of an objectusing the plurality of intermediate compressive measurements.
 2. Thecompressive sense imaging system of claim 1, wherein the processingdevice is further configured to: generate an uncompressed image of theobject from the plurality of compressive measurements using areconstruction basis matrix.
 3. The compressive sense imaging system ofclaim 1, wherein the processing device is further configured to:determine a kernel matrix based on properties of an aperture array ofaperture elements and a sensor, and, generate a sensing matrix using thekernel matrix and a reconstruction basis matrix.
 4. The compressivesense imaging system of claim 3, wherein the processing device isconfigured to: decompose the sensing matrix to generate the plurality ofsequence matrices.
 5. The compressive sense imaging system of claim 3,wherein the processing device is configured to: determine a sensitivityfunction for the sensor; determine at least one characteristic functionfor at least one of the aperture elements of the aperture array; computea kernel function by performing a convolution operation using thesensitivity function and the at least one characteristic function; and,determine the kernel matrix using the kernel function and an image. 6.The compressive sense imaging system of claim 2, wherein the processingdevice is further configured to: apply a sparsifying operator togenerate the uncompressed image of the object from the plurality ofcompressive measurements using the reconstruction basis matrix.
 7. Thecompressive sense imaging system of claim 1, further comprising: alensless camera unit including an aperture array of aperture elementsand a sensor for detecting light passing through the aperture elementsof the aperture array.
 8. The compressive sense imaging system of claim7, wherein the processing device is further configured to: selectiveenable or disable one or more of the aperture elements of the aperturearray based on at least one basis in a sequence matrix to acquire atleast one of the plurality of intermediate compressive measurementsduring a time period, the at least one of the plurality of intermediatecompressive measurements being determined based on an aggregated sum oflight detected by the sensor during the time period.
 9. The compressivesense imaging system of claim 7, wherein the aperture array is amicro-mirror array.
 10. The compressive sense imaging system of claim 7,wherein the aperture array is a LCD array.
 11. A method for compressivesense imaging, the method comprising: generating, using a processor, aplurality of sequence matrices; determining a plurality of intermediatecompressive measurements using the plurality of sequence matrices; and,generating a plurality of compressive measurements representing acompressed image of an object using the plurality of intermediatecompressive measurements.
 12. The method of claim 11, furthercomprising: generating an uncompressed image of the object from theplurality of compressive measurements using a reconstruction basismatrix.
 13. The method of claim 11, further comprising: determining akernel matrix based on properties of an aperture array of apertureelements and a sensor, and, generating a sensing matrix using the kernelmatrix and a reconstruction basis matrix.
 14. The method of claim 13,further comprising: decomposing the sensing matrix to generate theplurality of sequence matrices.
 15. The method of claim 13, furthercomprising: determining a sensitivity function for the sensor;determining at least one characteristic function for at least one of theaperture elements of the aperture array; computing a kernel function byperforming a convolution operation using the sensitivity function andthe at least one characteristic function; and, determining the kernelmatrix using the kernel function and an image.
 16. The method of claim12, further comprising: applying a sparsifying operator to generate theuncompressed image of the object from the plurality of compressivemeasurements using the reconstruction basis matrix.
 17. The method ofclaim 11, further comprising: selectively enabling or disabling one ormore aperture elements of an aperture array based on at least one basisin a sequence matrix to determine at least one of the plurality ofintermediate compressive measurements during a time period, the at leastone of the plurality of intermediate compressive measurements beingdetermined based on an aggregated sum of light detected by a sensorduring the time period.
 18. A non-transitory computer-readable mediumincluding one or more instructions for configuring a processor for:generating a plurality of sequence matrices; determining a plurality ofintermediate compressive measurements using the plurality of sequencematrices; and, generating a plurality of compressive measurementsrepresenting a compressed image of an object using the plurality ofintermediate compressive measurements.
 19. The non-transitorycomputer-readable medium of claim 18, including one or more instructionsfor further configuring the processor for: generating an uncompressedimage of the object from the plurality of compressive measurements usinga reconstruction basis matrix.
 20. The non-transitory computer-readablemedium of claim 18, including one or more instructions for furtherconfiguring the processor for: determining a kernel matrix based onproperties of an aperture array of aperture elements and a sensor;generating a sensing matrix using the kernel matrix and a reconstructionbasis matrix; and, decomposing the sensing matrix to generate theplurality of sequence matrices.